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ABSTRACT 



This research was an attempt to produce consistent 
course- level placement in college mathematics based on a predicted placement 
score in lieu of an actual placement score. Real data were used to determine 
the prediction equation and a second simple random sample from the same 
population was used to cross-validate the results. The consequences of this 
practical approach to placement has direct and immediate fiscal, spatial, and 
time management applications. A sample of 200 students was drawn from a 
population of 3,200 who took a mathematics placement test during the 
precollege session of a university. Placement test results, subsequent course 
results, American College Testing (ACT) Assessment test results, and high 
school percentile rank were available for these students. A multiple 
regression equation was developed using the total score based on the number 
of items correct as the dependent variable and the ACT math score and high 
school percentile rank as independent predictors. For cross-validation 
purposes, a second, independent data set of 200 students was obtained. 

Results of the comparison between actual and prediction based placement level 
for either data set agreed in approximately 55% of the cases, and for the 
remaining cases, a more conservative placement resulted in about 18% of the 
cases. In about 8%, the placement level was raised. The potential benefits of 
this approach at a large university are discussed. An appendix contains some 
comparative figures from the current and proposed scoring procedures. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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Abstract 



The need to have reliable and valid mathematics placement at the post- 
secondary level is imperative. The ramifications to students when institutions fail 
to meet this need range from minor academic frustration to potential failure in the 
course pf placement. The challenge to the institution is not limited to accurate 
placement but extends to consistent placement across time. 

This research attempts to produce consistent course level placement based 
on a predicted placement score in lieu of an actual placement score. Real data is 
used to determine the prediction equation and a second simple random sample 
from the same population is used to cross-validate the results The consequences 
of this practical approach to placement has direct and immediate fiscal, spatial, and 
time management applications. 
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Mathematics Placement Testing: Is it really necessary? 

Accurate placement is the culmination of many factors. Students arrive to 
institutions of higher learning from a variety of secondary schools. Different 
schools, with individual mission statements, offer different courses and programs. 
A major challenge facing institutions of higher learning is to sort through the 
variation among students, identify the communalities, and reasonably assess the 
content skills for each student. Due to practical financial, spatial, and time 
constraints, this assessment must be both quick as well as concise. Once the 
decision is made to stratify students into placement levels according to estimated 
skill level, it is the obligation of the institution to be consistent across students as 
well as across time. 

Unfortunately, measuring skill levels is not a simple task. Making 
predictions based on these measurements is even harder. Factors such as anxiety, 
frustration, fear, and disorientation often can and do influence test scores. 

Student^ may possess adequate skills and yet not perform well on a specific 
examination. Other students who do not possess the desired skills sometimes do, 
unfortunately, perform well. Both scenarios result in misplacement. 

The desired goals of placement testing are the obtainment of 
accurate estimates of skills and the subsequent correct placement of students into 
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academic levels. Each level offers a collection of mathematics courses, all 
homogeneous in terms of the minimal skills needed for a reasonable probability of 
success jn the selected course. For this estimate to be reliable, the testing 
instrument must be reliable. This is a necessary condition, though certainly not 
sufficient. The testing environment, time restrictions, if any, placed on the 
administration of the test, and even the administration itself, all can influence the 
reliability of the estimate. For the resulting placement to be valid, well-defined 
placement criteria must be established: subsequent results must be consistently and 
reasonably interpretable. 

The mathematics placement procedures at Ohio University are currently 
being assessed. 

This paper attempts to offer a viable option to actual test administration. 
Placement, in this author’s opinion, is an important advising tool. The question is 
not about placement per se; it is instead about placement testing-. Is it really 
necessary? 
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A brief history 

Ohio University currently requires all incoming students to take a 
mathematics placement test. This test is usually administered during the pre- 
college summer orientation, a program of which almost all traditional students 
partake. Tests are quickly scored and the results are readily available to facilitate 
Fall pre-registration, which also occurs during the pre-college session. 

There are areas of concern associated with the current placement test. This 
instrument consists of forty 4-option multiple choice items. These items span eight 
content areas, allowing for five items per content area. There are four placement 
levels of course offerings. Depending on a students placement score, they are 
placed into one of the possible placement levels. The current procedure for 
scoring and subsequent placement is based on the number of sections ‘passed A 
section (content area) is considered ‘passed’ if a student correctly answers at least 
three out of the five questions in that section. Under the current scoring/placement 
scheme, students with an equal number of sections passed may have very different 
raw score totals yet they will be placed into the same placement level. The ad hoc 
committee that is working on this placement issue at Ohio University agrees that a 
more consistent way of scoring is based on a linear, raw score total. The reader 
may refer to Appendices I and II for further details. 
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Item Analysis 



An item analysis performed in July, 1997, on the current placement test 
(N= 1,455 students) resulted in a Kuder-Richardson 21 (KR21) index of internal 
consistency of .855 with standard error of 2.940. The number of keyed items was 
forty: the mean score wps 21.925, the median score was 22.00 and the range was 
40. The standard deviation was reported to be 7.708 (variance 59.407). 

The analysis detected one item with negative discrimination and eleven 
items with undesirable difficulty indices, two of which had significant p-levels, 
a=.05. Furthermore, the negatively discriminating item demonstrated a point 
biserial correlation index of -.034 

The Spearman-Brown Prophecy from KR21 indicated that in order to 
obtain a reliability of .90 (with items such as these), this test would need to contain 
sixty two items. In this analysis, the number of missing and invalid responses was 

533. 

For many practical reasons, the ad hoc assessment committee within the 
mathematics department has recommended the creation of a new testing 
instrument. Due to the time investment necessary to design, write, pilot, revise, 
and polish a new placement test, an interim method is needed. The results of this 



paper provide one such option. 



Placement 



7 



Goals 



The goal of this research is to circumvent the administration of any 
particular placement test by predicting a placement raw score using a method of 
multiple regression and then placing students into stratified placement levels 
accordingly. The predicted placement scores would be used in lieu of actual 
scores. For purposes of comparison and generalizability, it is necessary to define 
placement criteria (cut scores) based on a raw score index. 

Resulting placement consistency will be measured with a cross-validation 
technique and a table of findings will be presented. More detailed tables will be 

provided in the appendices of this paper. 

The immediate implications of this work are the savings of time, space, and 
staffing associated with the annual testing of (approximately) three thousand 



individuals. 
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Method 



A simple random sample of 200 students 1 was drawn from a population of 
3,200 students who took the math placement test during their pre-college session, 
summer, 1997. Information offered by the Office of Institutional Research at Ohio 
University, Athens, Ohio, provided the following data for every student in the 
sample: percent correct pcore per section of the placement test, subsequent course 
assignment, ACT composite score, ACT English score, ACT reading score, ACT 
science reading score, ACT math score, and high school percentile rank. 

A multiple regression equation was developed using RAWTOTAL (total 
score based on the number of items correct) as the dependent variable and 
ACTMATH (ACT math score) and HSPCT (high school percentile rank) as 
independent predictors. Feasibility and underlying assumptions were checked as 
well as was the model design. A careful search for outliers and influential points 
was conducted. 

For cross-validation purposes, a second, independent data set (N2=200) 
was obtained. The regression equation was applied to the second set of data and 
nonparametric procedures were used for comparative purposes. The results of 
multiple pairwise comparisons between placements based on predictions with 
DATA1 and DATA2 were noted. Detailed results are included in the appendix. 
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Assumptions: 

(1) Linearity 

A linearity diagnostic assessment was conducted to determine the feasibility 
of using a multiple regression approach. 




The linear relationship between RAWTOTAL and ACTMATH is clear. In 
regards to a linear relationship between RAWTOTAL and HSPCT, the presence of 
a relative linear relationship can be argued in favor of, especially in light of the 
selective admissions policies in place at Ohio University and the understandable 
variation among high schools. 
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(2) Correlations: 

The Pearson correlation was computed for the dependent-independent pairs of 
variables, as well as for the intercorrelation between the predictors. 

The results can be seen in the following table: 





RAWTOTAL 


ACTMATH 


HSPCT 


Poareon 


RAWTOTAL 


1.000 




.767** 


.390** 


Correlation 


ACTMATH 


J67** 




1.000 


,259*< 




HSPCT 


.30O~ 




.259** 


1.000 


Sig. 


RAWTOTAL 






.000 


.000 


(2-tailed) 


ACTMATH 


.000 






.000 . 




HSPCT 


.000 




.000 




N 


RAWTOTAL 


200 




200 


200 




ACTMATH 


200 




200 


200 




HSPCT 


200 




200 


200 



**• Correlation Is significant at the 0.01 level (2-talled). 



Clearly there is a strong correlation between RAWTOTAL and 
ACTMATH (.767) as well as a moderate correlation between RAWTOTAL 
and HSPCT (.390). Furthermore, the intercorrelation between the two 
predictor variables is relatively low (.259). Multicollinearity does not seem to 
be a problem at this stage of the investigation. 
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(3) Homogeneity of Variance: 

As can be seen in the following scatterplot, the hypothesis of equal variances 
can be retained. 

Scatterplot 

Dependent Variable: RAWTOTAL 




Regression Standardized Predicted Value 



(4) Normality: Kolmogorov-Smimov Tests were computed for ACTMATH, 
HSPCT, and RAWTOTAL. In all three cases the hypotheses of normality 
were retained. 



Histogram 

Dependent Variable: RAWTOTAL 
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Normal P-P Plot o 1 Regression Stand; 
Dependent Variable: RAWTOTAL 
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The ENTER method of model selection was used. A multiple R value of .793 
was obtained; R 2 = .628 and adjusted R 2 = .624. This small amount of 
shrinkage is an indicator of cross-validation potential . Standard error of the 
estimate was calculated to be 4.53. A table of coefficients follows: 



Coeflidtnti . 
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1 (Constant) 


*15,020 


■ m 




m- 
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•IHtf 


' -10:754 












ACTMATH 


1.423 




.7M- 


15.175 


.000 


m 


m 


; 


.749 


m 


‘ 333 


1372 


HSPCT 


8359E42 


.020 




4.#? 


.000 


151 * 


.121 


■ .330 


309 


m - 


333 


1372 



a. Dependent Variable: RAWTOTAL 



The resulting regression equation can be written 
y = -15.020 + 1.43*ACTMATH + .08959*HSPCT. 

This is the regression equation that will be used on both DATA1 and 
DATA2 (the cross-validation sample) to calculate predicted raw scores from 
which placement assignments will be made. 
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y was calculated for every subject in DATA1 and all data were examined for 
outliers and influential points: 

(1) Mahalanobi’s Distance (D 2 ) was calculated and found to be 8.955. 

(2) Values of Cook’s distance (CD) were saved to the main data set and were 
carefully examined. None were found to exceed an upper bound of 1 

(3) Furthermore, outliers on Y were searched for via leverage elements. Three 
such points were found but none of these had CD>1 and hence, were 
dismissed from further scrutiny. 

Based on the newly derived values of y, placement levels were assigned via the 
following rawtotal placement criteria. 

Placement level raw score 

DV1 [0,18) 

PL1 [18,26) 

PL2 [26,32) 

PL3 [32,40] 
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The placement assignments for both the actual and the predicted 
placements were recoded into an ordinal scale for comparative purposes. 

Three pairwise comparisons were made on the following placements (for 
DATA1): actual placements based on the section-passed criteria, actual 
placements based on the rawtotal criteria, and predicted placements based on 
the rawtotal criteria. A Wilcoxon Signed Rank Test was used for all pairwise 
statistical comparisons. 

Upon request to the Office of Institutional Research at Ohio University 
and of the ad-hoc committee for placement of the Department of 
Mathematics, a second, independent simple random sample (N2 = 200) was 
drawn from the same population as the original sample of this study. The 
regression equation was applied to the second set of data. Variable encoding 
is explained in the first two of the following three tables. Tables one and two 
have been seperated on the basis of which data set (DATA1 or DATA2) was 
used. This was done simply for the sake of clarity. Results of the Wilcoxon 
Signed Ranks Tests follow in the third table. 

The reader should note that the key concept underlying the goals 
of this paper is consistency. Careful attention should be paid to Table 3. 

The pairwise comparisons between NASSIGN1->NASSIGN2 for DATA1 
and NASSIGN1->NASSIGN2 for DATA2 yield practically identical results. 
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Variable 




Table 1: DATA1 





1 • 8®B 1 ". .v-.. -: : MM 






Actual (consistent) placement by O.U. using DAT A 1 and old criteria. j 




Actual (consistent) placement using new placement criteria and DATA 1 . J 


&&$£■ fj. 

ti&gfflSSSNi 


Predicted placement using regression 1 , new criteria and DATA1 . j 



Table 2: DATA2 







jjaaSa^ 


Actual (consistent) placement by O.U. using DATA2 and old criteria. "1 




Actual (consistent) placement using DATA2 and new placement criteria. 




Predicted placement using regression 1 and DATA2. 


WmSm 


Predicted placement using regression! and DATA2. 
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Summary: 

The results of the comparison between actual and predicted-based 
placement level (for either data set) agreed in approximately 55% of the cases. 
Of the complementary 45%, recommended placement level was lowered in 
18% of the cases, meaning that a more conservative placement resulted from 
the regression method of placement. On the other hand, in about 8% of the 
cases, the placement level was raised. 

The same regression equation was used to calculate predicted raw 
scores for both (independent) sets of data (DATA1 and DATA2). Raw score 
placement criteria was applied in both instances. The findings described in the 
first paragraph therefore indicate that the placement results obtained from 
predicted raw scores are consistent with placement results obtained from 
actual testing. 

What we have observed here is intuitively consistent with reality at 
Ohio University. Experience has shown that a few students do feel that they 
have been placed ‘too low’ and request to be reassigned to a higher placement 
level. It also has been observed that a moderate number of students seek 
supplemental instruction and tutoring services early within the Fall quarter. It 
is this authors opinion that a majority of the students seeking help were simply 
misplaced under the current placement system, and that if a method is to err in 
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one direction or the other, it is better to err on the side on conservatism. 

The potential benefits to an institution (Ohio University, in this case) 
are great. If implemented, the regression method of placement would end the 
need to use pre-college time and personnel for test administration. Test 
security would no longer be an issue. Neither would be test revision and 
maintenance. Summer personnel as well as students would have placement 
information prior to pre-college, allowing for students to familiarize 
themselves with course options prior to meeting with academic advisors. 

Also, the timeliness of this information would allow for adjustments in the 
numbers of courses offered. Actual placement would remain an option for 
students/parents wanting a reconsideration of suggested placement level. 
Hence, some type of real placement test will still need to be available for these 
cases as well as for many non-traditional students for whom ACT scores 
and/or high school percentile rank are not available. 
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Appendix: 



I. Current Scoring Procedures: 

*a passed section consists of 3/5 items correct 
*there are four possible placement levels 



Level 


Number of Sections Passed 


Raw Score Range 


DV1 


0,1 


[0,19] 


PL1 


2,3,4 


[6,28] 


PL2 


5,6,7 


[15,37] 


PL3 


8 


[24,40] 



II. Recommended scoring procedure: 

* Eliminate the ‘sections-passes’ method 



Level 


Raw Score Range 


DV1 


[0,18) 


PL1 


[18,26) 


PL2 


[26,32) 


PL3 


[32,40] 
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Footnotes: 

1 This information was graciously provided by the Office of Institutional 
Research, Ohio University, Athens, Ohio. 
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